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APPARATUS AND METHOD FOR DYNAMICALLY CONTROLLING 
THE FRAME RATE OF VIDEO STREAMS 

This application claims the benefit of U.S. Provisional Application 
5 No. 60/090,814 filed on June 26, 1998, which is herein incorporated by 
reference. 



The invention relates generally to the field of digital multimedia 
communications over a network, e.g., computer networks. More specifically, 
10 the invention relates to a method of storing motion information for a 
sequence of images in different frame rates to effect dsmamic fi-ame rate 
control over a network. 

BACKGROUND OF THE DISCLOSURE 

15 The increasing demand for digital video/audio information presents 

an ever increasing problem of transmitting and/or storing an enormous 
amount of information. For example, the growth of the Internet and the 
proliferation of Internet browsers in recent years has enabled millions of 
users to access various information in a matter of seconds. Most of the 

20 information is currently in the form of text and/or still images and graphics. 
Generally, this type of information can be quickly downloaded and 
displayed by a user. More recently, audio and video clips or sequences have 
been added to numerous webs sites. Without an increase in bandwidth, 
accessing these clips requires more computational cycles and access time 

25 due to the sheer size of the image sequences. 

Two methods of accessing an image sequence across a network are 
available. First, the xiser can download the compressed bitstream in its 
entirety, and then decode and display the image sequence locally using the 
iiser's computer. One drawback of this method is the length of time that a 

30 user must wait while the image sequence is being downloaded. Another 
drawback is that the user may need to reserve a large storage space to 
acconunodate a large image sequence. 
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A second, and more practical, method is referred to as "video 
streaming" where the processes of downloading, decoding and displaying the 
image sequence can be performed simultaneously. Namely, each frame of a 
sequence is immediately decoded and displayed upon receipt by the user. 
5 However, video streaming poses a new set of challenges. GeneraUy, 

the pre-recorded original video sequence is compressed at a certain bit-rate. 
This bit rate may not coincide with the channel bit rate used by the user. 
Namely, users may download the image sequence at a different bit rate, 
depending on the available bandwidth, e.g., high user access time versus 
10 low user access time. Although the server can reduce the delivery rate of 
the image sequence to match the channel bit rate, such reduction wiU result 
in a "slow motion" display at the receiver. 

Alternatively, one solution to address varying bit rate is to partially 
decode the image sequence at the server, and then re-encode the image 
15 sequence using dififerent coding parameters, e.g., different quantizer scales, 
thereby resulting in a different bit-rate that coincides with the channel bit 
rate used by the user. This method has proven to be quite powerful, and 
accurate, for a reasonable range of bit-rates. Namely, this method 
effectively provides a 'Tmob" with which the user can vary the bit-raite of the 
20 image sequence. Unfortunately, as the bit rate decreases significantly, the 
effect of using this "knob" is a significant degradation in the spatial quaUty 
of the unage sequence. Furthermore, this "knob" would not be practical if 
the user wishes to maintain the spatial resolution at the expense of 
sacrificing the temporal resolution, e.g., reducing the firame rate from 30 
25 fi:«mes/second to 15 frames/second. 

Therefore, a need exists in the art for an apparatus and method for 
effecting dynamic firame rate control over a network. 

RTTMMARY OF THE IN VENTION 
30 An embodiment of the present invention provides a method for 

dynamically varying the frame rate of an image sequence. More 
specifically, in one embodiment, the image sequence is encoded and stored 
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at different frame rates (e.g., 30, 15, 10 and 5 fps). This embodiment 
increases the storage requirement for a server. 

In a second embodiment, only the motion information, e.g., motion 
vectors, for the other frame rates are stored. Namely, the image sequence is 
first encoded and stored at a preferred frame rate and then the same image 
sequence is encoded at one or more different frame rates. The encoding 
process at other frame rates can be performed off-line. However, only the 
motion information is saved and stored, e.g., in "motion files", for the other 
frame rates, thereby reducing the storage requirement of the server. Thus, 
when it is necessary to re-encode the image sequence to accommodate a 
different frame rate, motion vectors are obtained from a storage device, 
instead of being computed. This method enables the image sequence to be 
re-encoded at a different frame-rate in real-time. Additionally, the motion 
files are significantly smaller than the corresponding compressed image 
sequence. 



BRIEF DESCRIPTION OF THE DRAWINGS 
The teachings of the present invention can be readily understood by 

considering the following detailed description in conjxmction with thie 

accompanying drawings, in which: 

FIG. 1 illustrates a block diagram of a network environment having a 

server incorporating the dynamic frame rate control of the present 

invention; 

FIG- 2 illustrates a block diagram of an encoder incorporating the 
djntiamic frame rate control of the present invention; 

FIG. 3 illustrates a flowchart of a method for effecting the d3mamic 
frame rate control; 

FIG. 4 illustrates a block diagram of a motion file of the present 
invention; and 

FIG. 5 illustrates a block diagram of an encoding system of the 
present invention. 
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To facilitate understanding, identical reference numerals have been 
used, where possible, to designate identical elements that are common to 
the figures. 

5 DRTAILED P F-SnRIPTION 

FIG. 1 illustrates a block diagram of a network environment having a 
server 110 incorporating the dynamic frame rate control of the present 
invention. More specifically, a plurality of cHents or users 130 are able to 
access various image sequences stored on the server 130 via a 
10 communication or computer network 120. 

As discussed above, if the bit rate of the pre-recorded image sequence 
does not match the bit rate of the communication channel used by the dient, 
then the server can adjust the bit rate of the pre-recorded image sequence 
by decoding and re-encoding the image sequence in the bit rate of the 
15 communication channel used by the client. Unfortunately, since such real- 
time decoding/'re-encoding operation is computationally expensive, the 
server will Ukely be equipped with expensive special purpose hardware. 

More specifically, the computational cost is attributed to the 
extremely time-consuming process of estimating motion as employed in 
20 encoding standards such as MPEG and H.263 which employ motion 
estimation techniques. Some implementations may employ fiill motion 
search algorithms which generate very accurate motion information, but at 
the expense of incurring a very high cost in computational cycles. As such, 
these implementations are not practical when it is performed in real-time 
25 »«i»g software, thereby requiring the use of faster special purpose 

hardware. In addition, with these encoding standards, the server cannot 
arbitrarily drop frames in the compressed domain due to inter firame 
encoding. Thus, if the server is to re-encode the image sequence (video 
stream) at a different frame rate, the server will have to recompute the 
30 motion information, e.g., motion vectors at the new fi^e rate. 

The present invention discloses a dynamic frame rate control method 
without the use of special purpose hardware. Namely, FIG. 1 iUustrates a 



wo 00/01 151 PCTAJS99/14S10 




server 110 having an encoder 112, a decoder 114 and a storage 116. The 
encoder 112 is an encoder capable of performing motion estimation and 
retrieving motion information. As such, the encoder 112 can be 
implemented as an MPEG or H.263 compliant encoder. 
5 In operation, one or more image sequences are pre-encoded and 

stored in storage 116 at a particular bit rate and frame rate. If a client 130 
requests an image sequence from the server 110 that matches the bit rate 
and frame rate of the pre-recorded image sequence, then the server will 
simply forward the image sequence directly to the chent via the network 
10 without further processing. 

If the client requests an image sequence at a different frame rate 
from the frame rate in which the image sequence was encoded, then the 
server 110 must decode the encoded image sequence using the decoder 114. 
It should be noted that if the original image sequence is also stored in the 
15 server 110, then the decoding operation and the decoder 114 itself can be 
omitted all together, since the original image sequence is readily available. 

Once the original image sequence is obtained (either through 
decoding the encoded image sequence or retrieving the original image 
sequence from storage), the encoder 112 can now re-encode the image 
20 sequence at the new frame rate. However, the encoder 112 does not have to 
compute the motion information from the image sequence. Instead, the 
encoder 112 reads the motion information from a "^motion file" associated 
with the new frame rate, e,g., 25 fps, 20 fps, 15 fps, 10 fps, 5 fps and so on. 
Since the majority of the computational cycles in an encoding process is 
25 spent in generating motion information, e.g., motion vectors, encoder 112 
can now quickly re-encode the image sequence in real-time using the stored 
motion information to accommodate a frame rate as requested by a cUent. 

Effectively, a second knob is provided to a cUent for varying the frame 
rate of the image sequence. By reducing the frame-rate of the image 
30 sequence, the bandwidth will be reduced without ftuther loss in the spatial 
quality (temporal quality would be reduced instead). The present dynamic 
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frame rate control is further described in detail below with reference to 
FIGs. 3 and 4. 

FIG. 2 illustrates a block diagram of an exemplary encoder 200 
incorporating the dynamic frame rate control method of the present 
5 invention. Although the present invention is described below with reference 
to an MPEG compUant encoder, those skilled in the art will realize that the 
present invention can be adapted to other encoders that employ motion 
estimation technique, e.g., an H.263 compUant encoder. 

The encoder 200 comprises a motion estimation module 240, a motion 
10 compensation module 250. a rate contarol module 230, a DCT module 260. a 
quantization (Q) module 270, a variable length coding (VLC) module 280, a 
buffer 290, an inverse quantization (Q ') module 275, an inverse DCT (DCT 
transform module 265, a subtracter 215 a summer 255, and a storage 116 
for storing motion files. Although the encoder comprises a plurality of 
15 modules, those skilled in the art will reaUze that the functions performed by 
the various modules are not required to be isolated into separate modules as 
shown in FIG. 2. For example, the set of modules comprising the motion 
compensation module 250, inverse quantization module 275 and inverse 
DCT module 265 is generally known as an "embedded decoder^. 
20 FIG. 2 illustrates an input video image (image sequence) on signal 

path 210 which is digitized and represented as a luminance and two color 
difference signals (Y, C„ CJ in accordance with tiie MPEG standards. The 
digitized signal may optionally imdergo preprocessing such as format 
conversion for selecting an appropriate window, resolution and input 
25 format. 

The input video image on path 210 is received into motion estimation 
module 240 for estimating motion vectors. A motion vector is a two- 
dimensional vector which is used by motion compensation to provide an 
offset from the coordinate position of a block in the current picture to the 
30 coordinates in a reference frame. The reference frames can be a previous 
frame (P-frame), or previous and/or futiire frames (B-frames). The use of 
motion vectors greatly enhances image compression by reducing the amount 
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of information that is transmitted on a channel because only the changes 
between the current and reference frames are coded and transmitted. 

Alternatively, the present invention incorporates a storage 116 for 
storing motion files for an image sequence at various frame rates. Namely, 
5 if the current image sequence on path 210 has been previously encoded and 
the motion information for the image sequence is stored in storage 116, then 
the motion estimation process performed by the motion estimation module 
240 is bypassed and the associated motion vectors for a particular frame 
rate are simply read from a motion file stored in storage 116. This process 
10 greatly increases the speed with which the image sequence is encoded. 
However, if the image sequence was not previously encoded or the motion 
information for a desired frame rate was not previously generated, then the 
motion vectors are generated using motion estimation module 240. The 
resulting motion vectors can then be stored in a new motion file in storage 

15 lie. 

The motion vectors from the motion estimation module 240 or the 
storage 116 are received by the motion compensation module 250 for 
improving the efficiency of the prediction of sample values. Motion 
compensation involves a prediction that uses motion vectors to provide 
20 ofisets into the past and/or fiature reference frames containing previously 
decoded sample values that are used to form the prediction error. 

Furthermore, prior to performing motion compensation prediction for 
a given macroblock, a coding mode must be selected. In the area of coding 
mode decision, MPEG provides a pluraUty of different macroblock coding 
25 modes. Specifically, MPEG-2 provides macroblock coding modes which 
include intra mode, no motion compensation mode (No MC), 
frame/field/dual-prime motion compensation inter mode, 
forward/backward/average inter mode and field/frame DCT mode. 

Once a coding mode is selected, motion compensation module 250 
30 generates a motion compensated prediction (predicted image) on path 252 of 
the contents of the block based on past and/or future reference pictures. 
This motion compensated prediction on path 252 is subtracted via 



aNSOOCIO: <WO_00011S1A1J_> 



• PCT/US99/14S10 
8 

subtracter 215 from the video image on path 210 in the current macroblock 
to form an error signal or predictive residual signal on path 253. It should 
be noted that if a current frame is encoded as an I-frame, then the signal on 
path 253 is simply the original picture and not a predictive residual signal. 
5 The DCT module 260 then applies a forward discrete cosine 

transform process to each block of the predictive residual signal to produce a 
set of eight (8) by eight (8) block of DCT coefficients. The DCT basis 
function or subband decomposition permits effective use of psychovisual 
criteria which is important for the next step of quantization. It should be 
10 noted that other transforms can be employed, e.g., wavelet transform. 
The resulting 8x8 block of DCT coefficients is received by 
quantization module 270 and variable length coding (VLC) module 280 
where the DCT coefficients are quantized and coded. Finally, the data 
stream is received into a First In-First Out" (FIFO) buffer 290. In 
15 appUcations that involve a fixed-rate channel, a FIFO buffer is used to 
match the encoder output to the channel for smoothing the bit rate. Thus, 
the output signal of FIFO buffer 290 on path 295 is a compressed 
representation of the input video image on path 210. 

The rate control module 230 serves to monitor and adjust the bit rate 
20 of the data stream entering the FIFO buffer 290 to prevent overflow and 

underflow on the decoder side (wdthin a receiver or target storage device, not 
shown) after transmission of the data stream. Furthermore, the bit rate of 
the image sequence can be a4justed via rate control module 230, e.g., by 
changing the quantization scale. 
25 The resulting 8x8 block of quantized DCT coefficients from the 

quantization module 270 is also received by the inverse quantization 
module 275 via signal connection 272. At this stage, the encoder 
regenerates I-frames and P-frames of the input video image by decoding the 
data so that they are used as reference frames for subsequent encoding. 
30 The resulting dequantized 8x8 block of DCT coefficients are passed 

to the inverse DCT module 265 where inverse DCT is applied to each 
macroblock to produce the decoded error signal. This error signal is added 
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back to the prediction signal from the motion compensation modvile via 
summer 255 to produce a decoded reference picture (reconstructed image). 

FIG. 3 illustrates a flowchart of a method 300 for effecting the 
dynamic frame rate control of the present invention. More specifically, FIG. 
5 3 illustrates a method for allowing a client to d5mamically change the frame 
rate of an image sequence. 

Method 300 starts in step 305 and proceeds to step 310, where an 
image sequence is transferred to a client at a predefined fi^me rate. It 
should be noted that if the frame rate requested by the client does not 
10 match the frame rate of the pre-recorded image sequence, then step 310 is 
bypassed and method 300 proceeds to step 320. 

In step 320, method 300 queries whether the frame rate is changed. 
The frame rate of an image sequence can be changed for a number of 
different reasons, e.g., as requested by the client, or the bandwidth of the 
15 commimication channel has changed and is detected by the server. If the 
query is negatively answered, then method 310 returns to step 310 and 
continues sending the image sequence to the client at a predefined frame 
rate. If the query is affirmatively answered, then method 300 proceeds to 
step 330. 

20 In step 330, method 300 queries whether the image sequence is 

stored in the desired new frame rate. Namely, method 300 is inquiring 
whether there is a stored image sequence that was previously encoded at 
the desired firame rate. 

In one embodiment of the present invention, each image sequence is 

25 encoded at a plurality of different firame rates (stored information). For 
example, an original image sequence can be encoded in three different 
fi^e rates, such as 30 fps, 25 ^s and 20 £ps. If the frame rate is changed 
from 30 fps to 25 fps, the server simply reads the image sequence from a 
different file. However, this embodiment dictates a large storage 

30 requirement for the server. 

Returning to step 330, if the query is affirmatively answered, then 
method 300 proceeds to step 340, where the stored encoded unage sequence 
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at the new frame rate is retrieved from a storage, e.g., storage 116 of FIG. 1. 
Method 300 then returns to step 310 where the image sequence is sent to 
the client at the new predefined frame rate. If the query is negatively 
answered, then method 310 proceeds to step 350. where the original image 
5 sequence is encoded at the new frame rate using stored motion information, 
e.g., as stored in motion files. 

FIG. 4 illustrates a block diagram of an exemplary motion file 400 of 
the present invention. Motion file 400 comprises a start of frame code field 
410, a presence of motion information field 420, an optional mode 
10 information field 430 and the actual motion information 440. 

The start of frame code field 410 is a field that indicates the 
beginning of a frame, which is then followed by motion information for that 
frame, e.g.. a first fi^e start code field 410, foUowed by motion information 
for the first frame, a second frame start code field 410, followed by motion 
15 information for the second frame, and so on. Namely, the start of frame 
code field 410 serves as a demarcation for separating motion information of 

successive frames. 

To illustrates, if the present invention is implemented in conformance 
to the coding standard of H.263, the frame start code field is implemented 
20 as the 17 bit start of picture code for each frame that is present in all H.263 
encoded frames. This allows the real-time encodmg process to switch 
between frame rates in the middle of a transmission for example. 

The presence of motion information field 420 provides information to 
the encoder concerning block-based information. The following disclosure 
26 uses standard coding terminology; however, it should be understood that the 
term macroblock or block is intended to describe a block of pixels of any size 
or shape that is used for the basis of encoding. Broadly speaking, a 
-macroblock" could be as small as a single pixel, or as large as an entire 
video frame. Namely, since a given macroblock (MB) in H.263 can be coded 
30 in various forms (which will lead to a variable number of motion vectors for 
that MB) some information regarding the macroblock are added to the 
motion file to assist the encoder in using the motion information. 
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More specifically, the presence of motion information field 420 is 
implemented as a single bit per macroblock. A zero ("0*0 indicates that 
there are no motion vectors associated with a current MB (which would be 
the case if the MB is not encoded (COD = 1) or if it is encoded in INTRA 
5 mode). A one ('1') indicates that there is one motion vector for the ciorrent 
macroblock, where the motion vector is stored in the motion information 
field 440. 

Optionally, the mode information field 430 is provided to further 
distinguish between the COD = 1 case, and the INTRA mode. Namely, if 

10 the encoder is informed that no motion information is available for a current 
block, then the encoder must determine whether the absence of motion 
information is due to the lack of motion in the current block or the 
macroblock is being Intra coded. Generally, an encoder will perform a Sum 
of Absolute Difference (SAD) calculation when no motion vector is available 

15 for the purpose of determining whether the MB should be INTRA coded. 
Namely, the absolute difierence between each corresponding pixel value, 
e.g., within an area such as a macroblock between successive fi>ames is 
performed. Next, a sum of the all the absolute difference values is 
compared to a predicted SAD, which serves as a threshold to measure the 

20 degree of change in the macroblock. The Mode Decision is then selected 
accordingly, i.e.. Intra coding for substantial change and Inter coding for 
insubstantial change. 

Unfortunately, these SAD calculations are time consuming and will affect 
the performance of the real-time encoder. 

15 As such, the mode information field 430 provides an additional bit 

such that the combination of the presence of motion information field and 
the mode information field allows the implementation of a variable length 
code (one or two bits) in the motion file that can be used to indicate both the 
COD and the MB type (Intra or no motion). For example, a '0' would 

to indicate COD = 1, a '10' would indicate mode = INTRA and a '11' would 
indicate mode = INTER and presence of a motion vector. The 
implementation of the reason for absence of motion information field 430 
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increases the speed of the encoder at the expense of creating larger motion 
files. 

In the present invention, a motion file is generated for each frame 
rate of an image sequence. However, it should be noted that additional 
5 coding information can be inserted into the motion file such as Advanced 
Prediction (AP) mode and the PB-fi-ame mode as described in the H.263 
standard. However, as additional information is inserted into the motion 
file, the size of these motion files should be balanced so as not to impose an 
impractical increase in the storage reqviirement of the server. 
10 Returning to FIG. 3, in step 360. method 300 queries whether the 

transfer is completed. If the query is negatively answered, then method 310 
returns to step 310 and continues sending the image sequence to the client 
at a predefined fi-ame rate. If the query is affirmatively answered, then 
method 300 ends in step 370. 
15 In the present invention, the user can either specify the desired firame 

rate, or provide the target bit-rate to the server and have the server find an 
optimum between the firame rate and the quantization parameter. The 
server would then determine on a firame by firame basis if the firame is to be 
encoded or not. For fi-ames that need to be encoded and transmitted, it will 
20 open the motion file corresponding to the firame rate, and encode the firames 
accordingly. 

One of the advantages of the present invention is that since motion 
estimation is done off-line, a full search algorithm and a wide range of 
motion vectors can be used to obtain the optimum compression efficiency. 

25 FIG. 5 illustrates an encoding system 500 of the present invention. 

The encoding system comprises a general purpose computer 510 and various 
input/output devices 520. The general purpose computer comprises a 
central processing unit (CPU) 512, a memory 514, an encoder 516 and a 
decoder 518 for receiving and encoding a sequence of images. 

30 In the preferred embodiment, the encoder 516 is simply the encoder 

200 and the decoder 518 is simply the decoder 114 as discussed above. In 
feet, the entire server 110 can be implemented using the general purpose 



wo 00/01151 PCT/US99/14510 

-13- ~ 



computer 510. The encoder 516 and the decoder 518 can be physical devices 
which are coupled to the CPU 512 through a communication channel. 
Alternatively, the encoder 516 and the decoder 518 can be represented by a 
software application (or a combination of software and hardware, e.g., 
application specific integrated circuits (ASIC)) which is loaded from a 
storage device and resides in the memory 512 of the computer. As such, the 
encoder 200 of the present invention can be stored on a computer readable 
mediimi, e.g., a memory or storage device. In turn, the motion information, 
i.e., motion files and the encoded image sequence at various frame rates, 
which are generated by the encoder 516, can also be stored on a computer 
readable medium, e.g., RAM memory, magnetic or optical drive or diskette 
and the like. 

The computer 510 can be coupled to a plurahty of input and output 
devices 520, such as a keyboard, a mouse, a camera, a camcorder, a video 
monitor, any number of imaging devices or storage devices, including but 
not limited to, a tape drive, a floppy drive, a hard disk drive or a compact 
disk drive. The input devices serve to provide inputs to the computer for 
producing the encoded video bitstreams or to receive the sequence of video 
images from a storage device or an imaging device. 

Although various embodiments which incorporate the teachings of 
the present invention have been shown and described in detail herein, those 
skilled in the art can readily devise many other varied embodiments that 
still incorporate these teachings. 
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What is claimed is: 

1. A method for dynamicaUy controlhng the frame rate of an image 
sequence, said method comprising the steps of: 
5 a) transferring the image sequence at a first frame rate; 

b) receiving a request for a change from said first firame rate to a 

second frame rate; and 

c) transferring the image sequence at said second frame rate by 
retrieving stored information of said image sequence in accordance with 

10 said second frame rate. 

2. The method of claim 1, wherein said transferring step (c) comprises 
the step of transferring the image sequence at said second fi^e rate by 
retrieving a stored image sequence of said image sequence encoded at said 

15 second frame rate. 

3. The method of claim 1, wherein said transferring step (c) comprises 
the step of transferring the image sequence at said second frame rate by 
retrieving a stored motion information of said image sequence encoded at 

20 said second frame rate. 

4. The method of claim 3, wherein said retrieving step comprises the 
step of retrieving motion vectors from a motion file of said image sequence 
encoded at said second frame rate. 

25 

5 . A method for storing information of an image sequence to effect 
dynamic frame rate control, said method comprising the steps of: 

a) storing said image sequence encoded at a first frame rate; and 

b) storing the motion information of the image sequence encoded at 
30 least at a second firame rate. 
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6. A data structure stored on a computer readable medium comprising: 
a start of frame code field; and 
motion information field. 

5 7. The data structxire of claim 6, further comprising a presence of 
motion information field. 

8. The data structiire of claim 7, fiu1;her comprising a mode information 
field. 

10 

9. A server for d5niamically controlling the firame rate of an image 
sequence, said server comprising: 

a means for transferring the image sequence at a first firame rate; 

a means for receiving a request for a change fi-om said first firame 
15 rate to a second firame rate; and 

wherein said transferring means transfers the image sequence at said 
second firame rate by retrieving stored information of said image sequence 
in accordance with said second firame rate. 

20 10. The server of claim 9, wherein said stored information is a stored 
motion information of said image sequence encoded at said second firame 
rate. 
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